auxjoin qparser - a prunable segment-parallel auxiliary index join - #4749
auxjoin qparser - a prunable segment-parallel auxiliary index join#4749mkhludnev wants to merge 63 commits into
Conversation
significant TODOs
it doesn't show a gain anymore
- detects empty intersection column. sigh
- detects empty column when load them. sigh
github-actions
Bot
added
the
documentation
cleanup docs write single valued numerics DV
move into package-info.java
- removed eager write method - removed eager confirmation
load bigger segments first, and wait in the same order lazy confirming join leafs with bigger cardinality
dsmiley
left a comment
dsmiley
left a comment
There was a problem hiding this comment.
Exciting stuff here Mikhail!
|
|
||
| `auxIndexJoin` (which stands for Auxiliary Index Join) Query Parser is similar to xref:join-query-parser.adoc[], but uses a lazily written sidecar index for faster joins. | ||
|
|
||
| Instead of using Lucene's join utilities, the parser matches documents through a dedicated join index that is maintained alongside the core's main index. |
There was a problem hiding this comment.
I recall this index is in-memory and not on-disk. That's a key characteristic that shouldn't be omitted here. Simply insert "in-memory".
There was a problem hiding this comment.
Pardon for miscommunication. In the first version it's a plain FSDirectory.
Experiments with ByteBufferDirectory in the wait list so far.
|
|
||
| == Segment-level Parallelism | ||
|
|
||
| The algorithm utilizes multiple threads on _both_ sides if multiple threads are available. So, set xref:configuration-guide:configuring-solr-xml.adoc#indexSearcherExecutorThreads[indexSearcherExecutorThreads] to `-1` or `>0` in the `solr.xml` file. Note the emphasis on _both_ above: using the xref:common-query-parameters.adoc#multithreaded-parameter[`multiThreaded`] request parameter confines the inner (`from`-side) query to a single thread, limiting performance. |
There was a problem hiding this comment.
This confuses me. Why would using multiThreaded confine the from-side to a single-thread?
Would a user ever reasonably want that? If not, it might make sense to HTTP-400
There was a problem hiding this comment.
I can think of an edge case of many small segments causing many threads scheduling efforts. The current implementation is rather neat. I want to keep an option to control concurrency.
Co-authored-by: David Smiley <dsmiley@apache.org>
…ry-parser.adoc Co-authored-by: David Smiley <dsmiley@apache.org>
https://issues.apache.org/jira/browse/SOLR-18307
Description
Auxiliary Index Join QParserPlugin
Solution
Lazily write docvalues columns of
to_docnum[from_docnum]into sidecar index.Tests
Essential test coverage is provided.
Find benchmark (here)[https://github.com/mkhludnev/aijoin-benchmark]
Checklist
Please review the following and check all that apply:
mainbranch../gradlew check.